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INTRODUCTION:  Corps  of  Engineers  planners  have  been  required  to  consider  risk  and  uncer¬ 
tainty  since  the  Principles  and  Standards  Act  of  1972.  The  Actions  for  Change  are  increasing  the 
Corps  of  Engineers’  reliance  on  risk-informed  analysis  and  decision  making.  An  informal  review 
of  the  Corps’  experience  with  ecosystem  restoration  plans  has  shown  few,  if  any,  systematic 
efforts  to  address  the  often  significant  uncertainties  encountered  in  estimating  and  quantifying 
environmental  benefits.  Better  evaluation  of  these  uncertainties  and  consequent  project  risks  will 
improve  the  quality  of  ecosystem  restoration  planning  efforts. 

The  purpose  of  this  technical  note  is  to  encourage  the  systematic  and  routine  consideration  of 
uncertainty  when  estimating  environmental  benefits  of  ecosystem  restoration  projects.  To  that 
end,  this  note  will  address: 

•  Reasons  for  addressing  uncertainty. 

•  The  language  and  meaning  of  uncertainty. 

•  Appropriate  conditions  for  uncertainty  analysis. 

•  Methodologies  and  tools  appropriate  for  uncertainty  analysis. 

WHY  CONDUCT  UNCERTAINTY  ANALYSIS?  Uncertainty  analysis  is  done  to  improve  the 
quality  of  decisions  that  must  be  made  when  it  is  not  possible  to  have  all  the  desired  information. 
Uncertainty  analysis  adds  value  to  the  decision-making  process  for  the  Corps,  its  stakeholders, 
and  the  public.  It  enables  the  Corps  to  better  manage  ecosystem  restoration  projects  to  achieve 
desired  outcomes  for  national  ecosystem  restoration,  national  and  regional  economic  develop¬ 
ment,  and  other  social  effects. 

Burkes-Copes  et  al.  (in  review)  explored  the  state  of  the  science  for  incorporating  uncertainty  in 
ecosystem  restoration  projects.  One  of  their  main  findings  was  the  lack  of  uncertainty  analysis 
being  practiced  in  part  due  to  the  resistance  to  change  in  the  ways  “things  are  done.” 
Pappenberger  and  Beven  (2006)  found  that  a  significant  part  of  the  hydrological  and  hydraulic 
(H&H)  modeling  community  is  reluctant  to  embrace  the  estimation  of  uncertainty  in  their  work. 
They  explore  seven  common  arguments  for  resisting  uncertainty  estimation  and  find  each 
untenable.  The  rejected  reasons  for  opposing  uncertainty  analysis  in  H&H  work,  which  can  also 
be  applied  to  ecosystem  restoration  projects,  are: 
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•  Uncertainty  analysis  is  not  necessary  given  physically  realistic  models. 

•  Uncertainty  analysis  cannot  be  used  in  hydrological  and  hydraulic  hypothesis  testing. 

•  Uncertainty  (probability)  distributions  cannot  be  understood  by  policy  makers  and  the 
public. 

•  Uncertainty  analysis  cannot  be  incorporated  into  the  decision-making  process. 

•  Uncertainty  analysis  is  too  subjective. 

•  Uncertainty  analysis  is  too  difficult  to  perform. 

•  Uncertainty  does  not  really  matter  in  making  the  final  decision. 

Risk  analysis  is  an  essential  element  of  the  U.S.  Army  Corps  of  Engineers  Actions  for  Change, 
which  consists  of  the  following  four  themes: 

•  Comprehensive  systems  approach. 

•  Risk-informed  decision  making. 

•  Communication  of  risk  to  the  public. 

•  Professional  and  technical  expertise. 

Besides  addressing  the  actions  for  change,  there  are  a  number  of  additional  reasons  for  address¬ 
ing  uncertainty  in  estimating  environmental  benefits  for  ecosystem  restoration  projects,  including 
the  following: 

•  To  better  estimate  and  manage  risks.  Uncertainty  can  contribute  to  the  occurrence  of 
adverse  consequences  and  it  can  interfere  with  the  realization  of  potential  gains. 
Addressing  attendant  uncertainties  in  the  assessment,  implementation,  and  management 
of  project  alternatives  is  an  essential  part  of  good  risk  analysis. 

•  To  improve  the  quality  of  information  upon  which  decisions  are  based.  In  the  past,  proj¬ 
ect  outcome  forecasts  have  proven  to  be  wrong.  Assessing  uncertainty  in  the  candidate 
plans  enables  a  more  honest  and  realistic  comparison  of  plan  effects  and  their  likelihoods. 
Decision  makers  should  be  able  to  make  better  decisions  when  infonned  of  the  nature  and 
extent  of  uncertainty  in  analyses  supporting  that  decision. 

•  To  manage  stakeholder  expectations  of  project  outcomes.  Ecological  responses  to  a  resto¬ 
ration  project  are  often  difficult  to  predict.  By  assessing  the  uncertainty  in  project 
outcomes,  planners  can  guard  against  unrealistic  expectations  of  environmental  benefits. 

•  To  anticipate  the  possibility  of  undesirable  outcomes  and  avoid  surprise.  Ecological 
restoration  projects  can  sometimes  yield  undesirable  outcomes.  An  uncertainty  analysis 
can  help  planners  anticipate  the  possibility  that  these  outcomes  may  be  realized  and 
formulate  plans  to  manage  the  likelihood  that  undesirable  outcomes  are  realized. 

•  To  evaluate  the  sensitivity  of  project  outcomes.  Exploration  of  uncertainty  in  a  planning 
study  can  reveal  useful  insights  about  interdependencies  in  a  project. 

•  To  steer  projects  toward  desirable  outcomes.  The  Corps  and  its  partners  want  to  design 
“successful  projects”  that  will  achieve  targeted  outcomes.  Identifying  data  gaps  and 
uncertain  relationships  and  evaluating  management  alternatives  and  the  range  of  possible 
outcomes  is  part  of  implementing  an  adaptive  management  process. 

•  To  reduce  the  potential  propagation  of  random  error  or  bias  through  the  analysis.  A  care¬ 
ful,  systematic,  and  transparent  handling  of  the  relevant  uncertainties  can  reduce  the 
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likelihood  of  the  presence  of  undetected  cumulative  errors  in  model  results  and  metrics 
upon  which  decisions  are  based. 

•  To  ensure  safe  infrastructure.  Examining  uncertainty  in  the  performance  of  project  mea¬ 
sures  can  identify  and  help  mitigate  any  risks  associated  with  infrastructure  failure  or 
malfunction. 

•  To  help  control  costs.  There  are  often  uncertainties  in  estimating  the  cost  of  a  proposed 
ecological  restoration  project  alternative  as  well  as  estimating  its  ecological  benefits.  An 
understanding  of  these  uncertainties  can  help  improve  the  quality  of  net  benefit  estimates. 

•  To  comply  with  current  and  emerging  policy.  Corps  policy  (for  example,  EC  1 105-2-214 
(U.S.  Army  Corps  of  Engineers  (USACE)  1997)  and  ER  1105-2-100  (USACE  1990)) 
already  includes  requirements  to  account  for  uncertainty  in  water  resources  planning.  The 
Actions  for  Change  and  OMB ’s  growing  reliance  on  risk  analysis  methodologies  and 
concepts  ensure  that  this  reliance  on  uncertainty  analysis  will  only  increase  in  the  future. 

•  To  direct/target  research  efforts  (uncertainty  analyses  reveal  gaps  in  knowledge,  allowing 
researchers  to  focus  on  those  areas). 

RISK  ANALYSIS:  Risk  can  be  defined  as  the  probability  of  undesirable  consequences.  Risk 
analysis  is  an  analytical  process  that  includes  the  processes  of  risk  assessment,  risk  management, 
and  risk  communication  (Figure  1).  Risk  assessment  is  analytically  based,  whereas  risk  manage¬ 
ment  is  policy  and  preference  based.  Risk  communication  involves  the  interactive  exchange  of 
information  about  and  preferences  concerning  risk.  Risk  analysis  offers  a  systematic  approach  to 
decision  making  that  can  improve  the  quality  of  Corps  decisions.  Risk  is  often  assessed  in  terms 
of  potential  loss  of  life,  property  damage,  or  undesirable  ecological  outcomes  resulting  from 
natural  hazards  or  human  actions.  In  the  context  of  ecosystem  restoration,  project  risks  exist 
when  there  is  uncertainty  about  realizing  positive  net  benefits  from  an  investment. 


Figure  1.  Risk  analysis  involves  risk  assessment,  risk  management,  and 
risk  communication. 
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Both  the  probability  (likelihood)  and  consequences  (outcome)  of  a  decision  may  be  uncertain;  to 
distinguish  these  two  types  of  risks,  two  terms  are  adopted.  The  first  is  pure  risk.  This  is  the  risk 
of  loss  where  there  is  no  possibility  for  gain  and  it  is  consistent  with  the  traditional  definition  of 
risk  above.  It  most  often  involves  a  hazard  that  can  produce  adverse  effects.  In  water  resources 
these  include  natural  phenomena  (hazards)  that  cause  accidents  and  structural  failures.  Examples 
include  floods,  storms,  erosion,  droughts,  pollution,  water  quality  degradation,  infrastructure 
failure,  and  so  on.  Losses  occur  when  ecosystems,  resources,  people,  property,  or  other  assets  are 
exposed  to  the  hazard. 

Speculative  risk  or  project  management  risk  stems  from  the  chance  that  an  action  will  not 
achieve  the  predicted  or  desired  outcomes.  In  some  of  these  cases,  there  is  also  the  possibility 
that  an  action  will  achieve  outcomes  more  desired  than  those  predicted  as  well  as  less  preferred. 
Speculative  risk  refers  to  the  uncertain  outcome  of  actions  taken  to  alter  current  states  of  the 
world.  Failure  to  realize  desired  gains  is  one  fonn  of  loss.  Ecosystem  restoration  benefits  are 
primary  examples  of  desired  gains. 

For  risk  analysis  to  be  successful,  it  requires  analysts,  managers,  and  stakeholders  alike  to  give 
appropriate  attention  to  the  uncertainties  that  attend  the  decision  problem. 

WHAT  IS  UNCERTAINTY?  At  the  most  basic  level,  when  we  are  not  sure,  we  are  uncertain. 
The  literature  has  spawned  a  rich  language  to  express  this  simple  idea  in  more  elegant,  eloquent, 
and  sometimes  confusing  terms.  It  is  essential  to  understand  the  language  of  uncertainty  in  its 
current  usage  in  order  to  implement  best  practice  uncertainty  analysis.  Recognizing  that  other 
definitions  and  taxonomies  of  uncertainty  are  available  (e.g.,  Yoe  1996;  Regan  et  al.  2002), 
several  terms  are  defined  for  the  purposes  of  this  technical  note.  Two  basic  kinds  of  uncertainty 
are  defined  as  a  starting  point. 

Epistemic  uncertainty.  This  uncertainty  is  attributed  to  a  lack  of  knowledge  on  the  part  of  the 
observer  and  is  reducible  in  principle,  although  it  may  be  difficult  or  expensive  to  do  so.  It  arises 
from  incomplete  theory  and  lack  of  knowledge  about  the  behavior  of  the  system  under  study. 
(Epistemic  uncertainty  is  sometimes  called  subjective  uncertainty,  reducible  uncertainty,  and 
model  form  uncertainty.) 

Examples: 

•  Lack  of  experimental  data  to  characterize  new  materials  and  processes  (e.g., 
nanomaterials). 

•  Poor  understanding  of  the  linkages  between  water  quantity  and  habitat  desirability. 

•  Assuming  one  species  is  more  numerous  than  another  in  an  area  but  being  unable  to 
confirm  or  refute  this  assumption. 

Aleatory  uncertainty.  This  is  uncertainty  due  to  a  random  process.  It  is  attributed  to  the 
natural  variability  of  a  quantity  over  time  and/or  space  or  among  members  of  a  population.  It  can 
arise  because  of  natural,  unpredictable  variation  in  the  performance  of  the  system  under  study.  It 
is,  in  principle,  irreducible.  In  other  words,  the  variability  cannot  be  altered  by  obtaining  more 
information,  although  one’s  characterization  of  that  variability  might  change  given  additional 
information.  (Aleatory  uncertainty  is  sometimes  called  variability,  irreducible  uncertainty, 
stochastic  uncertainty,  or  random  uncertainty.) 
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Examples: 

•  Variation  in  daily  rainfall  and  runoff  over  the  next  30  years. 

•  Variation  in  fish  population  recruitment  due  to  seasonal  flooding. 

•  Variation  in  daily  mean  high  water  temperatures. 

Uncertainty  is  not  an  easy  concept  to  grasp  and  it  is  sometimes  useful  to  consider  the  difference 
between  closed  and  open  systems  when  characterizing  the  effects  of  uncertainty.  Both  kinds  of 
systems  occur  frequently  in  ecosystem  restoration  planning.  Consider  the  inherent  randomness  of 
the  natural  and  social  systems  with  which  planners  deal.  In  a  closed  system  it  is  possible  for 
analysis  to  bound  the  range  of  outcomes  (e.g.,  ecosystem  restoration  benefits)  and  offer  some 
estimate  of  the  likelihood  of  their  occurrence.  In  an  open  system,  planners  must  recognize  that 
water  bodies,  watersheds,  ecosystems,  and  human  decisions  are  not  only  characterized  by 
randomness,  but  also  difficult  to  predict  with  any  degree  of  confidence. 

Different  kinds  and  sources  of  uncertainty  will  most  likely  require  different  tools,  techniques, 
and  methodologies  to  best  address  them  in  the  planning  process.  In  addition  to  the  epistemic  and 
aleatory  distinctions  above  it  is  also  common  to  see  uncertainty  distinguished  by  its  source. 
Three  possible  sources  of  uncertainty  in  estimating  ecological  restoration  benefits  are  described 
here. 

Model  uncertainty.  This  is  uncertainty  about  the  models  used  to  estimate  the  benefits  of 
ecological  restoration  projects.  Model  uncertainty  can  arise  from  bias  or  imprecision  associated 
with  compromises  made  or  lack  of  adequate  knowledge  in  specifying  the  structure  and 
calibration  (parameter  estimation)  of  a  model. 

Examples: 

•  Index  model  formulas  and  the  relationships  amongst  variables  therein. 

•  Assumed  correlations  between  ecosystem  function  and  vegetation  composition  and 
structure. 

•  Cumulative  effects  (dependencies)  between  abiotic  and  biotic  model  components. 

Quantity  uncertainty.  This  is  uncertainty  about  which  value  to  use  for  an  input  variable  in  a 
model  to  estimate  ecological  restoration  outcomes.  This  category  also  includes  uncertainty  in 
measurements  from  a  population  of  interest.  This  source  is  the  practical  manifestation  of  aleatory 
uncertainty. 

Examples: 

•  The  number  of  bears  living  in  a  park. 

•  Mean  phosphorous  load  in  a  stream. 

•  Distance  between  water  control  structures. 

Scenario  uncertainty.  This  uncertainty  results  when  the  elements  of  a  scenario  or  their  rela¬ 
tionships  are  unknown  or  incomplete.  Scenarios  can  be  thought  of  as  the  stories  told  to  explain 
how  ecosystem  restoration  benefits  are  realized. 
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Examples: 

•  Misunderstanding  pathways  by  which  water  affects  a  habitat. 

•  The  effects  of  disturbance  on  an  ecosystem  subjected  to  urban  growth. 

•  The  response  of  the  ecosystem  to  climate  change. 


Figure  2  illustrates  the  types  of  uncertainty  and  includes  example  elements  such  as  stressors  and 
sources.  Scenario,  model,  and  quantity  (inputs)  uncertainty  are  likely  to  be  common  in  estimating 
ecosystem  restoration  benefits. 


SCENARIO 
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1 
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Figure  2.  Sources  of  variability  and  uncertainty  in  ecosystem  restoration. 


Scenarios  are  the  stories  told  to  explain  ecosystem  restoration  benefits.  In  planning  they  include 
without-  and  with-project  conditions,  historic  conditions,  and  perhaps  preferred  or  ideal  condi¬ 
tions.  Think  of  the  element  boxes  in  Figure  2  as  identifying  plot  lines  and  main  characters  in 
these  stories.  They  give  structure  to  the  nature  of  the  problems,  the  plan  effects  and  the  resulting 
benefits.  These  scenarios  in  turn  define  the  kinds  of  models  needed  to  estimate  ecosystem 
restoration  benefits.  They  also  identify  the  data  inputs  needed  to  run  those  models.  This  quantity 
uncertainty  is  discussed  in  some  detail  below. 

The  structure  of  the  model  includes  things  like  the  type  of  model  and  its  functional  form.  Models 
can  contain  varying  levels  of  detail;  more  realism  in  a  model  may  or  may  not  improve  the 
model’s  ability  to  support  good  decision  making.  Temporal,  spatial,  and  other  resolution  issues 
can  often  affect  model  outputs  significantly.  The  model’s  boundaries  will  assure  the  extent  to 
which  the  model  depicts  the  desired  scenarios.  Calibrating  and  validating  models  can  often  be  a 
challenge  when  data  are  inadequate  for  traditional  means  of  doing  so.  The  potential  cumulative 
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effects  of  these  sources  of  model  uncertainty  in  the  model’s  outputs  are  easy  to  envision  from 
this  figure.  As  a  matter  of  practice  model  uncertainty  has  not  received  the  attention  it  deserves. 
Models  are  rarely  considered  explicitly  as  a  potential  type  of  uncertainty  in  ecosystem  restoration 
benefit  estimation. 

QUANTITY  UNCERTAINTY:  Quantity  uncertainty  is  frequently  encountered  in  models  used 
to  estimate  the  benefits  of  ecological  restoration  projects.  Morgan  and  Henrion  (1990)  offer  a 
very  useful  taxonomy  for  those  seeking  to  understand  the  basic  types  of  uncertainty  in  models. 
Yoe  (1996)  describes  knowledge,  quantity,  and  model  uncertainty  for  environmental  invest¬ 
ments.  This  section  follows  Morgan  and  Henrion’s  work. 

Planning  requires  a  lot  of  information  that  is  frequently  gathered,  organized,  summarized, 
analyzed,  and  expressed  in  a  quantitative  manner.  If  there  are  167  acres  of  emerging  wetlands 
and  there  are  six  eagle  nests,  then  the  wood  duck’s  habitat  suitability  index  is  0.6.  The  cost  of  a 
duck  box  is  $320.  The  discount  rate  is  6.25%.  The  study  area  is  Cyber’s  Ranch.  The  quantities 
used  in  the  planning  process  are  frequently  a  major  source  of  uncertainty.  Having  a  way  to  think 
and  talk  about  uncertainty  is  critical  to  the  success  of  any  uncertainty  analysis. 

Table  1  presents  Morgan  and  Henrion’s  (1990)  classification  of  quantity  uncertainty  sources 
along  with  some  examples  and  methods  for  dealing  with  the  specific  type  of  quantity  uncer¬ 
tainty.  An  important  distinction  should  be  made  about  the  nature  of  these  quantities.  Only  the 
empirical  quantities  have  an  objectively  “true”  or  exact  value.  The  other  quantities  have 
subjective  values.  Hence,  it  is  most  appropriate  to  say  that  the  only  uncertain  quantities  are  the 
empirical  quantities.  For  all  other  quantities,  it  is  the  analyst  that  is  uncertain,  not  the  quantity 
itself.  This  technical  note  assumes  there  can  be  significant  uncertainty  about  the  best  or  preferred 
value  for  these  other  quantities.  The  significance  of  this  distinction  becomes  germane  when  one 
chooses  a  tool,  technique,  or  methodology  to  treat  the  uncertainty  appropriately. 


Table  1 

Types  of  Quantities  Encountered  in  the  Planning  Framework 

Quantity 

Example 

Treatment 

Empirical  quantities 

Birth  rate,  mortality  rate,  basal  area  of 
hardwoods,  Manning’s  n 

Probabilistic,  parametric 
variation 

Defined  constant 

Speed  of  light,  gallons  per  acre-foot, 
number  of  square  feet  in  an  acre 

Look  it  up 

Decision  variable 

Design  characteristics  of  decision 
alternatives  such  as  planting  densities  or 
beach  replenishment  rates 

Parametric  variation 

Value  parameter 

Discount  rate,  value  of  life,  weights  in  a 
multi-attribute  value  function 

Parametric  variation 

Index  variable 

References  to  locations  or  time  steps  in  a 
model 

Certain  by  definition 

Model  domain  parameter 

Study  area  boundaries,  planning  horizon, 
base  year 

Parametric  variation 

Outcome  criterion 

National  economic  development  benefits, 
benefit  cost  ratio,  habitat  units 

Depends  on  treatment  of  inputs 
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Empirical  quantities.  Empirical  quantities  may  be  the  most  common  quantities  encountered  in 
a  planning  study.  These  are  exact  values  that  are  unknown  to  the  analyst.  Where  these  quantities 
are  measurable  in  principle,  measurement  may  be  difficult  in  practice.  Examples  of  empirical 
quantities  at  a  restoration  site  might  include  the  basal  area  of  hardwoods,  the  density  of  nesting 
birds,  the  birth  rate  of  alligators,  or  the  number  of  seeds  produced  by  a  flower.  Although  there 
may  be  many  uncertain  empirical  quantities  in  a  planning  study,  it  is  often  sufficient  to  consider 
only  those  quantities  with  the  greatest  uncertainty  importance  (discussed  below). 

Defined  constants.  Defined  constants  are  fixed,  nevertheless  they  may  not  be  known  to  the 
analyst.  For  example,  there  are  43,560  ft2  in  1  acre  and  325,851  gal  of  water  in  1  acre-foot.  These 
constants  have  well  known  values  that  can  be  found  by  consulting  references. 

Decision  variables.  Decision  variables  are  quantities  over  which  the  planning  team  exerts 
direct  control.  Examples  include  the  density  at  which  marsh  grass  would  be  planted  under  a 
particular  revegetation  plan,  the  amount  of  sand  that  would  be  delivered  annually  under  a  beach 
replenishment  plan,  or  the  volume  of  water  that  would  be  diverted  each  month  to  nourish  and 
maintain  wetlands.  In  general,  these  are  the  characteristics  that  differentiate  decision  alternatives 
(plans)  from  one  another.  While  a  planner  may  be  uncertain  about  what  rate  of  beach  replenish¬ 
ment  would  actually  stabilize  the  beach,  the  replenishment  rate  is  a  chosen  characteristic  of  the 
proposed  alternative;  thus,  it  is  certain. 

Value  parameters.  These  parameters  describe  the  values  or  preferences  of  stakeholders,  the 
planning  team,  or  decision  makers.  Examples  include  the  rate  of  time  preference  for  money  (i.e., 
discount  rate)  or  the  weights  used  in  a  multicriteria  decision  analysis.  Planners  may  be  uncertain 
about  decision  makers’  preferences  and  the  value  these  parameters  should  take  in  a  decision 
model,  but  these  parameters  have  no  true  value  in  nature. 

Index  variables.  Index  variables  identify  elements  within  a  model,  such  as  a  location  within 
the  model’s  spatial  and  temporal  domain.  For  example,  a  point  in  time  can  be  referenced  as  a 
time  step  and  a  grid  cell  can  be  referenced  using  spatial  coordinates.  Index  variables  are  integral 
parts  of  a  model  and  are  not  considered  to  be  uncertain  variables. 

Model  domain  parameters.  These  parameters  describe  the  geographic  and  temporal 
boundaries  (domain)  of  a  model  and  the  resolution  of  its  inputs  and  outputs.  These  scale  charac¬ 
teristics  are  chosen  by  the  modeler  and  have  no  true  value  in  nature.  They  reflect  judgment 
regarding  the  model  domain  and  resolution  needed  to  estimate  ecological  restoration  benefits 
given  the  processes  described  in  the  model.  Not  all  representations  of  physical  or  ecological 
processes  can  be  easily  ported  across  scales  or  resolution,  and  considerable  effort  may  be 
required  to  alter  model  domain  parameters.  Uncertainty  about  domain  parameters  may  also  be 
considered  a  form  of  model  uncertainty.  The  sensitivity  of  model  output  to  the  choice  of  model 
domain  parameters  can  be  evaluated  by  treating  the  models  as  alternative  model  forms. 

Outcome  criteria.  Outcome  criteria,  also  termed  metrics,  are  variables  used  to  rank  or  measure 
the  desirability  of  possible  outcomes.  Examples  include  national  economic  development  benefits 
and  net  present  value.  When  evaluating  the  benefits  of  proposed  ecological  restoration  projects, 
outcome  criteria  might  be  expressed  in  ecological,  physical,  chemical,  or  social  terms.  Examples 
include  habitat  suitability  index  scores,  bank  migration  rates,  nutrient  concentrations,  and 
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stakeholder  desirability.  Uncertainty  in  the  output  of  a  model  is  evaluated  by  propagating 
uncertainty  from  the  input  variables  to  the  output  variables  using  one  of  several  different 
methods.  If  model  uncertainty  is  considered,  characterizations  of  uncertainty  in  the  outputs  of 
different  models  can  also  be  compared  and  evaluated. 

An  illustration.  There  are  often  large  uncertainties  in  the  outputs  of  ecological  models  such  as 
the  HSI  models  used  in  a  Habitat  Evaluation  Procedure.  These  uncertainties  can  be  large  relative 
to  the  ecological  effects  of  a  proposed  project.  If  so,  outcome  criteria  under  the  “with-”  and 
“without-”  project  conditions  may  not  be  statistically  different  from  each  another.  Figure  3  illu¬ 
strates  such  a  hypothetical  situation.  Panel  (a)  illustrates  uncertainty  in  modeled  ecological 
outcomes  (which  might  be  measured  in  thousands  of  habitat  units  on  the  x-axis)  under  a  “with-” 
and  “without-”  project  condition.  The  restoration  project  shifts  the  uncertainty  in  the  ecological 
outcome  to  the  right,  indicating  an  overall  improvement  in  ecological  condition.  However,  there 
is  so  much  uncertainty  in  these  ecological  outcomes  that  they  are  not  statistically  different  from 
one  another.  The  difference  in  expected  outcomes  under  the  distribution  A  is  small  relative  to  the 
uncertainty  in  HUs.  Panel  (b)  illustrates  uncertainty  in  the  ecological  benefit  of  the  proposed 
restoration  project.  The  ecological  benefit  is  the  difference  between  outcomes  under  with-  and 
without-project  conditions.  This  example  demonstrates  that  uncertainty  analysis  can  yield  a 
probability  distribution  on  the  benefit  measure  A.  The  benefits  are  uncertain,  but  results  show 
non-negative  benefits  of  the  restoration  effort  and  enable  decision  makers  to  evaluate  the 
probability  that  ecological  restoration  benefits  will  exceed  some  amount. 


Figure  3.  Uncertainty  in  the  benefit  measure  can  be  derived  from  uncertainty  in  model 
inputs  and  outputs.  Panel  (a)  illustrates  uncertainty  in  modeled  ecological 
outcomes  under  a  “with-”  and  “without-”  project  condition.  Panel  (b)  illustrates 
uncertainty  in  the  ecological  benefit  of  the  proposed  restoration  project. 


WHEN  IS  UNCERTAINTY  ANALYSIS  REQUIRED?  In  best  practice  planning  for  eco¬ 
system  restoration,  it  is  always  appropriate  to  examine  the  effects  of  uncertainty  on  decisions. 
The  extent  of  the  uncertainty  analysis  will  depend  on  the  amount  of  uncertainty  encountered  as 
well  as  the  consequences  of  a  wrong  decision.  Figure  4  illustrates  considerations  in  determining 
the  need  for  uncertainty  analysis. 
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Figure  4.  When  to  perform  uncertainty  analysis. 


The  most  rigorous  uncertainty  analysis  is  required  when  the  uncertainty  is  extensive  and  the 
consequences  of  making  a  wrong  decision  are  serious.  When  there  is  little  uncertainty  and  the 
consequences  of  a  wrong  decision  are  minor,  it  is  still  desirable  to  conduct  a  sensitivity  analysis 
that  describes  the  range  of  consequences,  if  only  to  verify  the  minor  consequences. 

In  the  intermediate  instances  when  there  is  much  uncertainty  in  an  analysis  but  the  consequences 
of  being  wrong  are  not  as  serious,  it  is  advisable  to  identify  the  key  uncertainties  and  address 
them.  When  the  uncertainties  are  fewer  in  number  but  have  more  serious  consequences,  it  is 
important  to  focus  on  them  to  examine  their  consequences  for  decision  making. 

It  is  always  appropriate  to  perform  uncertainty  analysis  for  ecosystem  restoration  projects.  The 
relevant  questions  are  how  extensive  or  rigorous  does  the  analysis  have  to  be  and  what  tech¬ 
niques  and  methodologies  are  available  for  each  situation.  The  next  section  identifies  some 
conceptual  approaches  to  uncertainty  analysis  as  well  as  some  of  the  techniques  and  method¬ 
ologies  available  to  the  analyst  that  are  suitable  for  uncertainty  analysis. 

UNCERTAINTY  ANALYSIS  TOOLBOX:  The  appropriate  techniques  for  uncertainty  analysis 
in  a  planning  investigation  depend  on  both  the  type  of  uncertainty  (epistemic,  aleatory,  scenario, 
model,  quantity)  and  the  cause  of  the  uncertainty.  It  may  be  convenient  to  think  of  aleatory 
uncertainty  as  out  there  in  the  world,  these  are  the  true  values,  the  exact  values  we  seek.  This  is 
contrasted  by  epistemic  uncertainty,  which  can  be  thought  of  as  inherent  in  the  analyst;  it  is  the 
analyst’s  uncertain  knowledge.  Different  tools  are  appropriate  for  the  various  kinds  of 
uncertainty. 
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Aleatory  uncertainty  of  empirical  quantities  is  perhaps  the  most  common  uncertainty  addressed 
in  estimating  ecosystem  restoration  benefits.  To  consider  this  uncertainty,  it  is  convenient  to 
follow  the  work  of  Morgan  and  Henrion  on  the  basic  causes  of  uncertainty  in  empirical  quan¬ 
tities  (Morgan  and  Henrion  1990;  Yoe  1996).  Probabilistic  methods  are  used  for  the  first  three 
causes  but  other  tools  like  parametric  variation  (intentionally  choosing  different  specific  values 
for  an  uncertain  value)  are  more  appropriate  for  other  causes.  These  causes  are: 

•  Random  error  and  statistical  variation. 

•  Randomness  and  unpredictability. 

•  Variability. 

•  Systematic  error  and  subjective  judgment. 

•  Linguistic  imprecision. 

•  Disagreement. 

•  Approximation. 

Framework.  An  anecdotal  review  of  selected  Corps  reports  indicates  that  “uncertainty 
analysis”  rarely  consists  of  more  than  a  brief  paragraph  or  two  discussing  the  topic  in  general  and 
nonspecific  ways  rather  than  the  actual  uncertainty  present  in  the  analysis.  The  best  way  to  begin 
to  address  uncertainty  in  a  systematic  and  transparent  manner  is  to  begin  simply.  The  simple 
steps  below  provide  a  skeletal  framework  for  any  uncertainty  analysis.  Analysts  should: 

•  Acknowledge  the  existence  of  uncertainty. 

•  Recognize  the  things  that  are  uncertain  (e.g.,  inputs,  assumptions). 

•  Identify  the  specific  uncertainties  of  concern  or  significance. 

•  Describe  the  nature  and  extent  of  the  uncertainty. 

•  Qualify  or  quantify  the  importance  of  uncertainty  for  decision  making. 

•  Communicate  the  extent  and  significance  of  uncertainty  to  decision  makers. 

•  Document  the  uncertainty  analysis. 

Toolbox. 

Narratives.  Approaches  to  uncertainty  analysis  can  be  divided  broadly  into  qualitative  and  quan¬ 
titative  techniques.  The  most  basic  way  to  address  uncertainty  is  via  a  narrative.  This  qualitative 
technique  is  simply  telling  the  story  of  the  key  uncertainties  and  their  significance  to  the  decision 
outcomes.  This  is  the  absolute  bare  minimum  effort  to  address  uncertainty. 

Although  very  simple  and  qualitative,  the  importance  of  this  technique  should  not  be  overlooked 
as  an  important  starting  point  for  uncertainty  analysis.  In  fact,  an  effective  narrative  needs  to 
accompany  every  uncertainty  analysis.  Not  everyone  will  need  to  understand  the  details  of  the 
uncertainty  analysis,  but  all  stakeholders  and  decision  makers  need  to  know  the  significance  of 
uncertainty  to  the  estimation  of  ecosystem  restoration  benefits  and  decision  making. 

Sensitivity  analysis.  The  most  common  analytical  technique  used  to  explore  the  significance  of 
uncertainty  is  sensitivity  analysis.  It  can  be  either  qualitative  or  quantitative.  Some  project  out¬ 
comes  and  decisions  are  sensitive  to  minor  changes  in  assumptions  and  input  values.  Thorough, 
rational  decision  making  requires  an  explicit  examination  of  such  sensitivities.  It  is  not  always 
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immediately  obvious  which  assumptions  and  uncertainties  may  affect  outputs,  conclusions,  and 
decisions  most.  The  purpose  of  sensitivity  analysis  is  to  systematically  make  this  determination. 

Sensitivity  analysis  is  a  systematic  investigation  of  model  parameters,  model  inputs,  assump¬ 
tions,  and  model  functional  forms.  The  cornerstones  of  sensitivity  analysis  include  challenging 
(and  changing)  assumptions,  along  with  parametric  variation  of  input  variable/parameter  values 
to  examine  these  effects  on  project  outputs. 

Sensitivity  analysis  is  used  to  increase  confidence  in  the  models  and  their  predictions.  It  provides 
understanding  of  how  model  outputs  respond  to  changes  in  inputs  (i.e.,  the  data  used),  model 
structures,  and  other  factors. 

Some  sensitivity  analysis  tools  include: 

•  Assumption  variation. 

•  Deterministic  one-at-a-time  analysis  of  each  factor. 

•  Deterministic  joint  analysis. 

•  Subjective  estimates  of  significant  threshold  values. 

•  Parametric  analysis  (using  a  range  of  values). 

•  Probabilistic  analysis  to  support  importance  analysis. 

Scenario  planning.  Scenario  planning  (Ralston  and  Wilson  2006)  is  an  appropriate  response 
when  there  are  relatively  few  but  relatively  important  epistemic  uncertainties  and  the  conse¬ 
quences  of  being  wrong  are  great.  Scenarios  are  narrative  descriptions  of  markedly  different 
plausible  alternative  futures.  Scenario  planning  allows  planners  to  consider  a  range  of  without-  or 
with-project  conditions,  each  of  which  is  dramatically  different  from  the  other  and  from  the 
current  operating  environment.  Rather  than  rely  on  a  single  “most  likely”  forecast,  planners  can 
compare  and  contrast  alternative  opinions  on  how  the  future  may  evolve  (e.g.,  IPCC  climate 
change  scenarios). 

Scenario  analysis.  Scenario  analysis  is  a  term  often  used  to  describe  a  varied  bundle  of  tools 
used  in  risk  and  uncertainty  analyses.  Scenarios  are  the  stories  we  tell  about  problems,  plans,  and 
their  effects.  The  Corps  relies  on  without-  and  with-condition  scenario  analysis  in  its  planning 
work  and  other  scenarios  are  often  considered.  Detenninistic  scenario  analysis  examines  specific 
scenarios  used  to  explore  the  range  of  effects  uncertainty  can  have  on  decision  criteria.  Some 
common  deterministic  scenarios  include  a  worst  case,  best  case,  most  likely,  locally  preferred, 
nonstructural,  and  no  action  scenarios. 

Probabilistic  scenario  analysis  is  one  of  the  most  common  and  powerful  quantitative  responses  to 
empirical  quantity  uncertainty.  Because  of  the  presence  of  variability  and  uncertainty  in  so  many 
ecosystem  restoration  studies  there  are  often  an  infinite  number  of  possible  future  scenarios.  It  is 
not  possible  to  describe  them  all,  but  some  of  them  may  be  important  to  the  decision  process. 

Probability  is  the  language  of  variability  and  uncertainty  and  it  can  be  incorporated  into  scenario 
analysis  using  such  techniques  as  the  Monte  Carlo  process,  interval  analysis,  fuzzy  set  theory, 
possibility  theory,  evidence  theory  (Dempster-Shafer),  and  imprecise  probability  theory.  Most  of 
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these  theories  are  in  an  early  stage  of  development  relative  to  classical  probability  theory  (i.e., 
Monte  Carlo  processes  and  Bayesian  estimation). 

Uncertain  quantities  can  be  represented  as  random  variables.  Random  variables  can  be  described 
using  frequency  distributions,  statistical  variances,  confidence  intervals,  and  probability  distri¬ 
butions.  Point  estimates  of  values  in  a  model  may  be  replaced  with  one  of  these  representations 
of  uncertainty  and  then  a  technique  such  as  Monte  Carlo  simulation  is  used  to  propagate  the 
uncertainty  of  the  outcomes  of  interest  through  multiple  calculations  of  the  model.  Figure  5  illu¬ 
strates  three  ways  of  describing  uncertainty  in  a  random  variable. 


Figure  5.  Three  ways  to  describe  uncertainty  in  a  random  variable.  Panel  (a)  presents  a  histogram  (often 
used  to  model  empirical  data),  Panel  (b)  presents  a  probability  density  function  (PDF),  and 
Panel  (c)  presents  a  cumulative  distribution  function. 


Adaptive  management.  Uncertainty  analysis  gives  rise  to  the  development  of  adaptive  manage¬ 
ment  strategies.  Adaptive  management  recognizes  at  the  beginning  that  uncertainty  is  inherent  in 
any  natural  system  and  seeks  to  minimize  this  uncertainty  by  learning  about  the  system  being 
managed  and  reducing  epistemic  uncertainties  that  are  recognized  at  the  time  the  action  is  taken. 
In  the  adaptive  management  process,  one  chooses  an  action,  monitors  the  effects  of  the  action, 
and  adjusts  the  action  based  on  the  monitoring  results  (Satterstrom  et  al.  2005). 

Premise  sets.  Premise  sets  were  used  by  the  Corps  to  address  significant  uncertainties  encoun¬ 
tered  in  the  aftermath  of  the  Mount  Saint  Helens  eruption.  Analysts  identify  the  range  of  possible 
outcomes  as  well  the  assumptions  and  other  things  one  must  believe  for  each  outcome  to  be 
realized.  Decision  makers  then  identify  their  view  of  the  uncertain  future  by  choosing  from  these 
sets  of  premises  and  their  consequences;  analysts  prepare  the  set  decision-makers  consider  most 
likely. 

IWR  plan.  The  IWR  Planning  Suite  is  decision  support  software  developed  by  and  for  the 
Corps’  ecosystem  restoration  planning  efforts  (Robinson  et  al.  1995;  Rogers  et  al.  2006). 
Although  originally  developed  to  assist  plan  formulation  by  combining  user-defined  solutions  to 
planning  problems  and  calculating  the  effects  of  each  combination  (or  "plan"),  it  now  includes 
enhancements  for  addressing  uncertainty.  These  are  most  nearly  described  as  probabilistic 
scenario  analysis  of  the  cost-effectiveness  of  the  various  plans  and  incremental  cost  analyses. 
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Other  techniques.  Several  other  techniques  can  be  used  to  address  uncertainty.  Uncertainty 
rankings  can  be  used  to  rank  benefit  estimates  from  the  least  to  most  uncertain.  Confidence  rank¬ 
ings  enable  analysts  to  express  their  degree  of  confidence  about  analyses.  Qualitative  scales, 
defined  by  the  analyst,  such  as  very  certain,  reasonably  certain,  moderately  certain,  moderately 
uncertain,  and  very  uncertain  can  be  used  to  place  the  analysis  in  a  context  for  decision  makers  to 
consider. 

The  minimax  regret  criteria  approach  (Yoe  1992)  is  designed  to  aid  decision  making  under 
uncertainty  that  estimates  the  opportunity  cost  (regret)  associated  with  each  possible  course  of 
action.  The  decision-maker  selects  the  activity  that  minimizes  the  maximum  regret,  or  loss. 
Regret  is  measured  as  the  difference  between  the  best  and  worst  possible  payoff  for  each  option. 

WHAT  CAN  BE  DONE  WITH  RESULTS?  The  ultimate  purpose  of  uncertainty  analysis  is  to 
improve  the  quality  of  ecosystem  restoration  project  decisions.  This  in  essence  means  managing 
the  risks  associated  with  natural  systems,  natural  hazards,  and  decision  making  under  uncertain 
conditions.  The  goal  of  risk  management  is  to  improve  the  capacity  for  decision  participants  to 
understand  and  then  make  decisions  in  consideration  of  natural  system  variability  and  analytical 
uncertainty,  so  as  to  increase  the  likelihood  that  the  intended  gains  will  be  realized.  Risk 
management,  then,  can  be  thought  of  as  planning,  analyzing,  organizing,  implementing,  and 
monitoring  efforts  to  control  the  effects  of  risk,  both  pure  and  speculative,  on  the  Corps’  Civil 
Works  Program.  An  effectively  conducted  and  communicated  uncertainty  analysis  is  an  essential 
component  of  good  risk  management. 

ADDITIONAL  INFORMATION:  This  technical  note  was  prepared  by  the  authors  under  the 
aegis  of  Environmental  Benefits  Analysis  at  the  Environmental  Laboratory,  U.S.  Army  Engineer 
Research  and  Development  Center,  Vicksburg,  MS.  Invaluable  input  was  provided  by  Brad 
Thompson,  Jodi  Staebell,  and  Shawn  Komlos.  Environmental  benefits  analysis  is  actively 
sponsored  by  the  Ecosystem  Management  and  Restoration  Research  Program  (EMRRP).  For 
more  infonnation  on  EMRRP,  please  consult  http://el.erdc.usace.army.mil/emirp/emirp.html  or 
contact  the  Program  Manager,  Mr.  Glenn  Rhett,  at  Glenn.G.Rhett@usace.army.mil.  This 
technical  note  should  be  cited  as  follows: 

Yoe,  C.  E.,  K.  Burks-Copes,  M.  T.  Schultz,  and  B.  C.  Suedel.  2009.  Addressing 
risk  and  uncertainty  in  ecological  restoration  projects.  EMRRP  Technical  Notes 
Collection.  ERDC  TN  EMRRP-ER-13.  Vicksburg,  MS:  U.S.  Army  Engineer 
Research  and  Development  Center,  http://el.erdc.usace.army.mil/emrrp/ 
ernrrp.html 
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